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^ (57) Abstract: A system and method for providing enhanced computer-aided reading are disclosed. A rapid serial visual presenta- 
tion (RSVP) based text file includes portions having text and at least one code portion (3) that links to a particular text portion (4) 
one or more audio files (4) or other files (4) that stimulate a reader's senses. When a computer-based reading device reads the next 
^ portion (2) and corresponding code portion (3), the audio file (4) is retrieved and executed substantially in synchronicity with the 
^ presentation of the text in the text portion (2). In this way, the reader is presented with an enhanced reading experience. 
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SYSTEM AND METHOD FOR RAPID SERIAL VISUAL 
PRESENTATION WITH AUDIO 

BACKGROUND OF THE INVENTION 

Technical Field of the Invention 

The present invention relates to rapid serial visual presentation (RSVP) with 
audio and/or other sensory information, and particularly to providing and utilizing 
RSVP-based text files with audio information. 

Background of the Invention 

Reading text, such as an electronic book, from a conventionally sized computer 
monitor involves much of the same activity as in reading printed material. Several 
lines of text are simultaneously presented on the computer monitor and the reader 
reads each line of text before scrolling or paging to the next set of lines of text. 

A problem exists, however, when attempting to read a relatively large amount 
of text from a relatively small display, such as a display screen for a PDA or 
Smartphone. In this case, the reader cannot efficiently read a relatively large amount 
of text by utilizing the scrolling and other features utilized in reading from a large 
computer screen. Consequently, users of devices having small display screens are 
forced to transport the text to a computer or other device having a conventionally sized 
computer screen in order to read the text. 

Advancements have been made to allow users of devices haying small displays 
to effectively read large amounts of text. Rapid serial visual presentation, or RSVP, 
involves the brief display of consecutive words at a fixed location. Typically, the rate 
of presentation per word is set by the experimenter or the reader, and each word is 
shown for an equivalent amount of time. In most of the research with RSVP, the 
display duration of each word is very brief, as implied by the word "rapid" in the 
acronym. Regardless of the rate of presentation, the functional relevance of RSVP lies 
in the fact that a small amount of text, such as a word or small number of words, is 
presented to the reader at any one time. This allows for use of small screens, like 
screens used in PDAs or wireless telephones. By presenting text as a stream of text 
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to a reader such that only a small amount of text is presented to the reader at a time, 
users of PDAs, wireless telephones and other devices may easily and efficiently read 
relatively large amount of text. 

Like reading printed material, computer-aided reading is typically a silent 
5 activity. However, with the advancements in computer technology, computers are now 
capable of providing a wide variety of multi-media related tasks. It would be a benefit 
to incorporate multi-media and other audio-visual features to computer-aided reading 
so as to enhance the reading experience. 

1 0 SUMMARY OF THE INVENTION 

The present invention overcomes shortcomings in prior computer-aided 
reading systems and solves a significant need for such a system that enhances the 
traditional silent reading experience. According to a preferred embodiment of the 
present invention, a reading file is provided that includes one or more text portions 
1 5 having text therein that forms a stream of text that is presentable to a reader in groups 
of one or more words at a time. The reading file further includes at least one sense 
code portion associated with one or more words in an adj acent text portion. The value 
of the sense code portion identifies a sensory presentation file, such as an audio file. 
When the text from the adjacent text portion is visually presented to the reader on the 

2 0 visual text display unit 23, the identified sensory presentation file is executed so that 

execution of the identified sensorypresentation file occurs substantially synchronously 
with the text visual presentation. In this way, the Reading experience is enhanced to 
include sounds or other sensory presentations that are related to the particular text 
presented to the reader, such as taste, smell and tactile presentations. 
25 An embodiment of the present invention includes a display device, such as a 

portable, hand-held communication device having a small display, including a 
processing unit that reads the text portion and a sense code portion associated 
therewith, identifies the sensory presentation file associated with the text portion, 
determines the amount of the sensory presentation file to be presented to the reader 

3 0 based upon the selected reading speed, and presents the sensory presentation file 

substantially synchronously with the presentation of the associated text portion. The 
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display device may include a wizard that creates the reading file from a text file, such 
as an RSVP-based text file, by identifying words in the text file that are related to a 
particular sound or other sense, and placing a tag code to each identified word whose 
value identifies an audio file. The audio file is subsequently executed when the tagged 
5 word is presented to the reader. 

BRIEF DESCRIPTION OF THE DRAWINGS 

A more complete understanding of the system and method of the present 
invention may be obtained by reference to the following Detailed Description when 
1 0 taken in conjunction with the accompanying Drawings wherein: 

Figure 1 is a portion of a reading file and related sensory presentation file in 
accordance with a preferred embodiment of the present invention; 

Figure 2 is a function block diagram of a display device for presenting the 
information in the reading file of Figure 1; 
1 5 Figure 3 is a flow chart illustrating an operation of the display device according 

to a preferred embodiment of the present invention; and 

Figure 4 is a flow chart illustrating another operation of the display device in 
creating the reading file, in accordance with a preferred embodiment of the present 
invention. 

20 

DETAILED DESCRIPTION OF THE PREFERRED EXEMPLARY 
EMBODIMENTS 

The present invention will now be described more fully hereinafter with 
reference to the accompanying drawings in which a preferred embodiment of the 
25 invention is shown. This invention may, however, be embodied in many different 
forms and should not be construed as being limited to the embodiment set forth herein. 
Rather, the embodiment is provided so that this disclosure will be thorough and 
complete, and will fully convey the scope of the invention to those skilled in the art. 
Referring to Figures 1-2, there is shown a signal format and corresponding 
3 0 device for presenting text-based information to a reader. The text-based information, 
forming a reading file 1, is presented as a stream of words that is displayed on a screen 
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one or a few words at a time. In a preferred embodiment of the present invention, the 
reading file 1 utilizes the rapid serial visual presentation (RSVP) technique. It is 
understood, however, that the reading file 1 may utilize other techniques for presenting 
a stream of text on a display one or a few words at a time. 
5 Reading file 1 includes text portions 2 having therein text that forms a stream 

of text that is presented on a display in one or a few words at a time, as explained 
above. Reading file 1 further includes one or more sense code portions 3 having 
therein a code that identifies a particular sensory presentation file 4. Sense code 
portion 3 is sized to suitably identify one of a plurality of sensory presentation files 4. 

10 A sense code portion 3 is located within reading file 1 adjacent and/or proximally to 
a text portion 2 with which sense code portion 3 is associated. Each sense code 
portion 3 links a particular sensory presentation file 4 to a word or group of words in 
an adjacent text portion 2, and is available to be executed substantially simultaneously 
with the presentation of the associated reading file 1 . In this way, tibte execution of a 

15 sensory presentation file stimulates a human sense in synchronicity with the 
presentation of the associated text to the reader. 

In a preferred embodiment of the present invention, sensory presentation files 
4 include audio files that are audibly presented to a reader when played. Audio files 
4 may be MP3 audio files, but it is understood that audio files 4 may be other audio 

20 files. 

It is understood that reading file 1 and sensory presentation files 4 may be 
combined to form a single multi-media file and/or experience file. In this scenario, 
each sense code portion 3 may link an executable sensory portion of the multi-media 
file to a specific text portion 2 thereof. 
25 In other embodiments of the present invention, sensory presentation files 4 may 

instead include files that stimulate other human senses when played and/or executed. 

For instance, sensory presentation files 4 may include video (MPEG) or picture 
(JPEG) files that are presented simultaneously with the particular text to which sensory 
presentation files 4 are linked. Alternatively, sensory presentation files 4 may include 
3 0 files that stimulate a reader's sense of taste, touch or smell when executed. 

In presenting a stream of text to a reader, it is understood that the speed of the 
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text stream is controllable by the reader. In order to synchronously present the 
execution of a sensory presentation file 4 with the presentation of the associated text 
linked thereto, it is necessary to control the length of play of each sensory presentation 
file 4 accordingly. Specifically, if sensory presentation files 4 are linked so that the 
5 duration of execution thereof corresponds to a relative slow reading pace, such as 1 00 
wpm, it is desired to shorten the duration of execution of sensory presentation files 4 
for faster reading speeds. In a preferred embodiment of the present invention, each 
sensory presentation file 4 is partitioned into one or more regions 5 of sensory 
information. Each region 5 is prioritized relative to each other. The regions 5 of 

10 sensory information having the relatively low priorities assigned thereto are not 
presented to the reader at higher reading speeds. As a result, both the content and 
duration of a sensory presentation file 4 are synchronized to the text linked thereto by 
controlling the duration of the sensory presentation file based upon the speed at which 
text is presented to the reader. 

15 In a preferred embodiment of the present invention, each region 5 of sensory 

information of sensory presentation file 4 includes a priority code 6 having a value that 
identifies the relative ranking and/or priority of region 5. It is understood, however, 
that priority codes 6 may be located elsewhere, such as in sense code portion 3 of 
reading file 1 . 

20 A preferred embodiment of the present invention includes a display device 20 

(Figure 2) that is adapted to receive reading file 1 and present text and other sensory 
information to the reader. Display device 20 may include a processing unit 21 for 
reading reading file 1 stored in memory 22 and presenting text therein as a stream of 
text on display unit 23 one word or a group of words at a time. In addition, display 

25 device 20 includes one or more sensory presentation units 24 for presenting the 
contents of sensory presentation files 4 to the reader. Sensory presentation files 4 are 
stored in memory 22. A sensory presentation unit 24, for example, may be a speaker 
for playing/executing an audio file 4. 

Display device 20 may be a portable, hand-held communication device, such 

30 as a Smartphone or PDA. Accordingly, display device 20 may further include a 
receiving unit 25 that receives reading file 1 and sensory presentation files 4 linked 
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thereto as one or more wireless signals for storage in memory 22. For example, 
reading file 1 and corresponding sensory presentation files 4 are downloaded 
substantially simultaneously from a source (not shown) using the wireless application 
protocol (WAP). 

5 When retrieving and decoding reading file 1 from memory 22, processing unit 

21 parses text from text portions 2 for presenting on display unit 23 in groups of one 
or more words. Processing unit 21 decodes the code value in a sense code control 
portion 3 and thereby identifies a sensory presentation file 4 to execute substantially 
simultaneously with the display of text to which the identified sensory presentation file 

10 4 is linked. Processing unit 21 determines the duration of and/or number of regions 
5 of sensory information of the identified sensory presentation file 4 to execute and/or 
present to the reader, based in part upon the speed at which the stream of text is 
displayed on display unit 23. The (reading) speed is provided to processing unit 21 
from the reader/user of display device 20. 

15 As stated above, both the reading file 1 and corresponding sensory presentation 

files 4 are downloaded to display device 20. This scenario allows for a composer to 
add sense code control portions 3 to an RSVP-based text file to create reading file 1 . 
In another preferred embodiment of the present invention, a text-to-audio wizard 27 
may be utilized to substantially automatically add sense code control portions 3 to 

20 reading file 1. In particular, application wizard 27, stored in memory 22, causes 
processing unit 21 to analyze an RSVP-based text file and identify text therein that can 
be presented audibly. For example, wizard 27 may cause processing unit 2 1 to identify 
verbs that may be associated with a sound. Wizard 27 and/or processing unit 2 1 place 
a tag and/or code control portion 3 on and/or near the identified text in the RSVP- 

2 5 based text file that identify an audio file 4 in a bank 28 of audio files stored in memory 

22. When the RSVP-based text file (having code control portions 3 therein) is 
subsequently retrieved from memory 22 for presentation to the reader and the audio 
file 4 identified in the sense code control portion 3 exists in memory 22, the identified 
audio file 4 is executed substantially synchronously with the presentation of the tagged 

3 0 text 

The operation of the preferred embodiment(s) of the present invention will be 
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described with reference to Figures 3 and 4. Initially, it is assumed that a reading file 
1 is downloaded into memory 22 of display device 20 at step 30 together with sensory 
presentation files associated with reading file 1. Next, the reader provides to 
processing unit 21 at step 3 1 the desired reading speed at which the stream of text from 
5 reading file 1 is to be presented on visual text display unit 23. Text portion 2 and a 
corresponding sense code control portion 3 of reading file 1 are retrieved from 
memory 22 and decoded at step 32. Processing unit 21 prepares text from retrieved 
text portion 22 for presentation and identifies sensory presentation file 4 from control 
code portion 3 at step 33. Based upon the reading speed provided during step 31, 

10 processing unit 21 determines the duration and/or the number of regions 5 of the 
identified sensory presentation file 4 to be executed at step 34. The prepared text is 
presented on display unit 23 and the region(s) 5 of the identified sensory presentation 
file 4 is executed substantially simultaneously with the presentation of the prepared 
text at step 35. Processing unit 2 1 then repeats steps 32-35 until the entire contents of 

1 5 reading file 1 has been retrieved and presented to the reader or until the reader signals 
to processing unit 21 to stop the presentation. 

Ia the instance in which display device 20 includes a text-to-audio software 
wizard 27, the operation of preparing reading file 1 so as to include one or more links 
to a sensory presentation file 4 will be described with reference to Figure 4. At step 

20 40, an RS VP-based text file is read by processing unit 21 and words identified 
therefrom that may be associated with a sound. Next, each identified word is tagged 
at step 41 by adding a control code thereto tljat indicate a particular sensory 
presentation file 4 in bank 28 of sensory presentation files 4. Upon a match existing 
between each tagged control code and a corresponding sensory presentation file 4 

2 5 existing in bank 28, the corresponding sensory presentation files 4 are thereby linked 
to the RSVP-based text file. 

The invention being thus described, it will be obvious that the same maybe 
varied in many ways. Such variations are not to be regarded as a departure from the 
spirit and scope of the invention, and all such modifications as would be obvious to 

30 one skilled in the art are intended to be included within the scope of the following 
claims. 
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What is claimed is: 

1 . A reading file interpretable by a computer-aided reading device, comprising: 
a plurality of text portions, each text portion including text so as to form a 

stream of text that is presentable to a reader a group of one or more words at a time; 
5 and 

at least one sense code portion associated with at least one text portion, the at 
least one sense code portion having a code therein that identifies an executable sensory 
presentation file and links the executable sensory presentation file to text in the text 
portion associated therewith. 

10 

2. The reading file of claim 1, wherein: 

the sensory presentation file comprises an audio file. 

3. The reading file of claim 1, further comprising: 

15 a plurality of sense code portions, each sense code portion being associated 

with a distinct text portion and including a code therein that identifies an executable 
sensory presentation file. 

4. The reading file of claim 1, wherein: 

20 the sensory presentation file comprises an audio file of a sound that 

corresponds to at least one word in the text portion associated therewith. 

5. A method of presenting information from a reading file to a reader, 
comprising the steps of: 

2 5 retrieving a first portion of the reading file having text information therein; 

retrieving a second portion of the reading file associated with the first portion 
and including a code therein; 

determining a sensory presentation file identified by the code in the second 
portion; 

3 0 visually presenting the text from the first portion; and 

executing the sensory presentation file substantially synchronously with the 
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step of visually presenting the text from the first portion. 

6. The method of claim 5, wherein: 

the sensory presentation file comprises an audio file. 

7. The method of claim 5, further comprising the step of: 
receiving a desired reading speed from the reader; and 

wherein the method of executing the sensory presentation file comprises the 
step of determining a duration of the sensory presentation file execution based upon 
the desired reading speed. 

8. The method of claim 7, wherein the step of determining comprises the step 

of: 

determining regions that partition the sensory presentation file; 

determining priority information corresponding to each region of the sensory 
presentation file; and 

executing one or more regions of the sensory presentation file having the 
higher priority information based upon the desired reading speed. 

9. The method of claim 5, further comprising the step of: 

initially receiving the reading file and related presentation files using the 
wireless application protocol. % 

10. A computer program product for presenting text from a reading file on a 
display device including a processing unit, memory and a display unit, the computer 
program product including instructions for: 

retrieving from memory in the display device a first portion of the reading file 
having text information therein; 

retrieving from memory in the display device a second portion of the reading 
file associated with the first portion and including a code therein; 
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determining a sensory presentation file stored in memoryidentifiedbythe code 
in the second portion; 

visually presenting the text from the first portion on the display unit; and 
executing the sensory presentation file substantially synchronously with the 
text from the first portion being presented on the display unit. 

1 1 . The computer program product of claim 10, wherein: 
the display device comprises a speaker; and 

the sensory presentation file comprises an audio file so that executing the 
10 sensory presentation file comprises playing the audio file using the speaker. 

12. The computer program product of claim 10, further including instructions 

for: 

receiving a desired reading speed from the reader at the processing unit; and 
15 wherein executing the sensory presentation file comprises determining a 

duration of the sensory presentation file execution based upon the desired reading 
speed. 

13. The computer program product of claim 10, wherein the instructions for 
2 0 executing comprises instructions for: 

determining regions that partition the sensory presentation file; 
determining priority information corresponding to each region of the sensory 
presentation file; and 

executing one or more regions of the sensory presentation file having only the 
2 5 higher priority information based upon the desired reading speed. 

14. The computer program product of claim 1 0, further including instructions 

for: 

initially receiving the reading file using a wireless protocol. 

30 



a 

5 



WO 01/80027 



PCT/SE01/00861 



-11- 



1 5 . The computer program product of claim 1 0, further including instructions 

5 for: 

determining regions that partition the sensory presentation file and determining 
priority information corresponding to each region of the sensory presentation file based 
upon contents of the sensory presentation file. 

10 1 6. A method of creating a reading file for displaying a stream of text from an 

electronic device, comprising the steps of: 

identifying words in an RSVP-based text file corresponding to a sound; 
for each identified word, determining a code value therefor that links an audio 
file to the identified word in the RSVP-based text file, a sound generated by executing 
1 5 the audio file being substantially the same as the sound corresponding to the identified 
word; and 

adding each code value to the RSVP-based text file. 

17. The method of claim 16, wherein: 

2 0 each code value is placed within the RSVP-based text file substantially 

pioximally to the corresponding identified word. 

18. A device for presenting a stream of text to a reader, comprising: 
a display unit for displaying text thereon; 

2 5 memory for storing a reading file having a plurality of text portions containing 

text information and at least one sense code portion containing a code therein; and 

a processing unit for reading the reading file, processing the text portions for 
presenting the text therein as a stream to the display unit, identifying the sense code 
portion and the text portion corresponding thereto, retrieving a sensory presentation 

3 0 file from the memory identified by the code in the sense code portion, and executing 
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the sensory presentation file substantially synchronously with the presentation of the 
text from the corresponding text portion. 

19. The device of claim 18, further comprising: 
5 a speaker; 

wherein the sensory presentation file comprises an audio file, execution of the 
audio file being played on the speaker. 



20. The device of claim 19, wherein: 
1 0 the processing unit determines an amount of the audio file to be played on the 

speaker based upon the amount of time the text from the corresponding text portion 
is displayed on the display unit. 



15 



21. The device of claim 19, wherein: 

the device comprises a wireless telephone. 
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